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ABSTRACT 

In this study the impact of computer immersion on performance of school leavers Senior 
Certificate mathematics scores was investigated across 31 schools in the EMDC East education 
district of Cape Town, South Africa by comparing performance between two groups: a control and 
an experimental group. The experimental group (14 high schools) had access to computers since 
2001 while the control schools received computers between 2006 and early 2007. That is, the 
experimental schools could be expected to be more immersed in computer technology than the 
control schools. Findings indicated that there was no significant difference between the final 
Senior Certificate mathematics results of the schools with the computers and those without; no 
significant change in the results after the Khanya labs were installed; no significant change in the 
percentage of pupils that passed Senior Certificate Mathematics; and no significant change in 
Higher Grade Maths enrolment rates. This finding points to the need for caution in the 
implementation of ICT’s into schools as a potential panacea for mathematical failure in our 
context. Recommendations for further qualitative work to provide a more nuanced picture of 
computer usage should be made. 
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INTRODUCTION 

The rationale for this research is twofold: on the one hand South Africa currently faces a crisis in 
mathematics education, which has seen it placed last 1 in the Third International Mathematics and 
Science Study (Howie 2001; Howie et al., 2000; Global competitiveness report 2013, Evan 2013). 
Despite improvements 2 in the Senior Certificate 3 results over the 19 year period since the first 
democratic elections, Chisholm (2004) indicates that the quality of primary education remains 
poor in South Africa, especially in under-resourced schools, where grade 6 students, for example, 
perform 3 years below grade level (Taylor, Muller and Vinjevold 2003). Recent research (Evans, 
2013) indicates that South Africa is ranked second last in the world in terms of mathematics and 
science proficiency. In a bid to address this problem and build technological capacity in the 
country, especially in disadvantaged schools, the Western Cape Education Department (WCED) 
has committed itself to the integration of computer technology into schools under the Khanya 
project initiative. The introduction of computer software to improve mathematical performance is 


1 The international average score for 38 countries was 487 points; South Africa achieved a total of 275 
points. 

2 The extent to which these results represent real gains has been a much debated issue in the local media, 
see for example Jonathan Jansen’s ‘Matric quick fixes miss the mark’ published in the Sunday Times 
4/1/2004 which sparked a flurry of commentary. 

3 This is a school leaving examination, colloquially referred to as the ‘matric exam’, which students write in 
their final year of schooling. Obtaining a matric endorsement enables students to proceed to university. 
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informed by a well-established relationship between learning outcomes and learning resources 
(Schollar 2001). The assumption underlying the implementation of computer-based technology, 
such as mathematics software, into schools in South Africa is that the technology will help to 
develop autonomous learners, who are both mathematically and technologically literate and, in 
doing so, will help to bridge the digital divide that continues to grow in South Africa (Department 
of Education 1996; 2000). While there are some studies (Howell and Lundall 1997, 2002; 
Organisation for Economic Co-operation and Development (OECD) 2003; Howie, Muller and 
Paterson 2005; Hardman 2008) that investigate the implementation of computers into schools in 
South Africa and while the South African Institute for Distance Education (SAIDE) (2004) 
research uses case study methodology, an extensive 4 review of the research in the field in South 
Africa has not revealed any comprehensive case studies that investigate the impact of computers 
on Senior Certificate results. What is disturbing is that international benchmarking, in the form of 
the Trends in International Mathematics and Science Study (TIMSS), shows that South African 
pupils are way below their peers internationally when it comes to Mathematics (and, indeed, 
Science). 

A further indicator that all is not well in the sphere of Mathematical education in South Africa is 
the ongoing poor Matric 5 Mathematics enrolment and results. The first issue is the very low 
number of pupils that chose Mathematics as a subject in the Grade 10-12 band under the 
previous curriculum, where it was optional (all pupils in Grades 10-12 now have to take either 
Mathematics or Maths Literacy). For example, in 2007 only 61.5% of enrolled Matrics chose 
Mathematics as a subject (Department of Education 2009). The second issue is the number of 
pupils that passed Mathematics at Matric Level: in 2007 less than a third (32.5%) of all Matric 
pupils gained a pass in Mathematics at some level, with only 4.5% passing at Higher Grade level; 
the level accepted by universities as sufficient for study in the science or technology fields 
(Department of Education 2009). 

Various attempts have been made by governmental and non-governmental departments and 
organisations to ameliorate this alarming situation. In particular, with reference to this article, the 
Western Cape Education Department (WCED) established the Khanya Project in April 2001 “to 
determine the contribution that technology could make towards addressing the increasing 
shortage of educator capacity in schools. With many skilled educators leaving the profession, 
fewer ones entering it, and AIDS already starting to take a significant toll amongst educators, it 
was necessary to explore alternatives. One of these alternatives is to use technology, already 
being used extensively in other disciplines, as an aid to augment teaching capacity” (van Wyk 

2002 p.21). 

The Khanya Business Plan, version 4.1 and dated 26 March 2002, described the “very ambitious 
goal” of the project to be: “By the start of the 2012 academic year, every educator in every school 
of the Western Cape will be empowered to use appropriate and available technology to deliver 
curriculum to each and every learner in the Western Cape.” (van Wyk 2002, p.10). The emphasis 
of the Khanya Project is “to use technology as a teaching aid, hence to improve curriculum 
delivery.” (van Wyk 2002, p.11). 

In order to meet its goals, the Khanya Project has since its inception been rolling out technology 
(computers, computer laboratories, numeracy and literacy software, ICT teacher training and 
technical support) to some of the most disadvantaged schools around the Western Cape in a 
series of phases, termed ‘waves’. By 2007 there had been seven such waves. The mathematical 


4 This search focused on 1) published journal articles, 2) government reports and 3) NGO websites. 

5 Matric is the name given to set of school-leaving examinations written by Grade 12 pupils in South Africa. 
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software systems provided to the high schools in the Khanya Project are one of two South 
African-produced systems: MasterMaths (2007) or CAMI Maths (2009). Both are examples of 
what is termed Computer Assisted Instruction (CAI), defined as involving the use of computers 
and computer software to provide drill exercises and tutorials (Kirkpatrick and Cuban 1998). 
While current research in CAI focuses on authentic learning as opposed to mere drill and practice 
exercises and tutorials, the Khanya project's chosen software is best described as drill and 
practice and not as promoting authentic learning (Hardman, 2008). 

It is against this background of poor Mathematics enrolment in the last three years of high school, 
disappointingly low Matric Mathematics pass rates, and the Khanya intervention as one means of 
ameliorating these issues that this research was undertaken in one education district of the 
Western Cape Province of South Africa, in order to see if the access to the Khanya computers 
has had any significant impact on Matric Mathematics results and Higher Grade enrolment. We 
hypothesised that computers would impact positively on student performance. 

Research on a Link between Computer Usage and Academic Attainment - Computers and 
Mathematics Performance in Developed Nations 

The impact of computers on academic (particularly mathematical) attainment is a much studied 
topic in the developed world, with numerous studies emanating from research in the United 
Kingdom - see Watson (1993); BECTA (2001); Higgins (2001); Harrison et al. (2002); Harrison, 
Lunzer, Tymms, Fitz-Gibbon and Restorick (2004) - and the United States - see Christmann, 
Badgett and Lucking (1997); and Tienken and Wilson (2007). 

A recent landscape review of the impact of ICT in schools (Condie and Munro 2007) analysed 
over 350 varied literature sources published since 2000 that are related to the impact of ICT in 
UK schools. Their highly equivocal conclusion is the following: “the evidence of the impact of ICT 
on attainment is, as yet, inconsistent, although there are some indications that in some contexts, 
with some pupils, in some disciplines, attainment has been enhanced. There is not a sufficient 
body of evidence in any of these areas, however, to draw firm conclusions in terms of explanatory 
or contributory factors” (Condie and Munro 2007, p. 29). 

All these studies were originated from relatively wealthy European or North American nations. 
This is unfortunate but unavoidable as it is only in those countries where computers have been 
used in schools for many years, and for which there are decades of research into ICT impact. It is 
questionable whether all the findings outlined will be transferable to impoverished schools in a 
city at the foot of Africa. 

Research showing Positive Impact on Particular Strands of Mathematics 

Amidst this ambivalence, a number of studies have shown ICT to have produced positive effects 
in various strands of Mathematics: 

• Clements (2002) reported that the use of Logo helped pupils to develop higher levels of 
geometric thinking and to learn geometric concepts and skills; while Forsythe (2007) 
discovered that the use of dynamic geometry software (specifically, Geometer’s 
Sketchpad) aided geometric understanding. 

• Raghavan, Sartoris and Glaser (1997) showed that 6 th grade pupils who were taught 
concepts of area and volume using a computer-based programme performed better 
overall than 8 th grade pupils taught traditionally, especially on the more complex 
problems. 
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• Various studies have shown the positive effect of computer algebra software and tutoring 
programmes on algebra exam scores - see Koedinger et al. (1997), Shaw, Jean and 
Peck (1997), Stephens and Konvalina (1999) and Barrow et al. (2007). 

• Cox and Nikolopoulou (1997) and Hennessy (2000) illustrated the benefits of the use of 
computers in developing data-handling skills, extrapolation and interpolation. 

One topic-based piece of research that shows clearly that using computers produced worse 
results than more traditional teaching methods is that of Wong and Evans (2007). Their study, 
involving Year 5 pupils in Sydney, Australia, showed that pen and paper instruction (PPI) 
methods was better than a computer based computer software package (Back to Basics Maths 
Multiplication) in improving pupils' basic multiplication fact recall. 

Research on Computers and Mathematics Performance in Developing Nations 

When one reads literature reviews on ICT and attainment, like that of Cox et al. (2003) or Condie 
and Munro (2007), one is struck by the large body of research that has been done in developed 
countries, and the almost complete lack of (reported) research in developing nations, like South 
Africa. It has been shown that the level of effectiveness of computers in improving mathematical 
understanding is context-dependent (Noss and Pachler 1999) and thus this dearth of research 
into this topic in disadvantaged schools (but particularly high schools) around the world is a 
serious omission that the current paper attempts to address. 

The only significant peer-reviewed exceptions to this general paucity are: 

i. The Chilean national Enlaces (links) programme, as reported in Hinostroza et al. (2002) 
and Somekh (2007). This programme, which provided computers, educational software 
and the like to the vast majority of Chilean schools was shown to have a number of 
peripheral positive impacts, yet case studies of some of the schools in the Enlaces 
programme have not provided evidence of measurable gains on traditional national 
students' assessment tests. 

ii. An Indian study on computer-assisted learning (CAL) by Banerjee et al. (2005) involving 
an investigation of 15 000 students. They studied a two year long CAL programme for 
over 15 000 children in Grades 2 to 4 in the city of Vadodara, and found that mid- and 
post-intervention test scores showed that the CAL programme had had a substantial, 
statistically significant positive effect on Mathematics achievements, increasing 
Mathematics scores by 0.35 standard deviations in the first year of the intervention and 
0.47 in the second year. It was equally effective for all pupils, from the strongest to the 
weakest academically. 

iii. A Turkish study on the use of dynamic geometry software, as reported by Isaksal and 
Askar (2005), reported no statistically significant impact of computer use. This study, 
involving 7 th grade pupils from one school instructed in a variety of mathematical topics, 
showed that there was no mean significant difference between the scores of the 
Autograph- and traditionally taught groups. 

iv. A South African study by Louw et al. (2008) into the effect of the use of MasterMaths 6 on 
Matric mathematics results in a sample of ten schools in the Western Cape province of 
South Africa; five experimental schools and five sample schools, found that that there “is 
only equivocal support for the effectiveness of the [Khanya] intervention” (Louw et al. 


6 MasterMaths is a Mathematics software programme that is used by many Khanya schools and which 
provides tutoring support 
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2008 p. 49), but that “the amount of time that learners spent [using MasterMaths] was 
significantly correlated with an improvement in mathematics performance” (Louw et al. 
2008 p. 49). There “is a clear, but not conclusive indication that the Khanya intervention 
improves mathematics performance in Grade 12 learners” (Louw et al., 2008 p. 49). 

It should be noted that only one of these studies, that of Louw et al. (2008), was completed in 
what in South African terms would be considered a high school. 

As mentioned above, the quantitative aspect of my research will replicate much of the statistical 
work of Louw et al. (2008), though in different schools and with more recent data. What will be 
interesting to see is whether the impact of the computers on mathematical performance has 
increased or decreased a few years after Louw et al’s study, bearing in mind that in some Khanya 
schools the 2007 matric students will have had access to computers and software for 5 years; 
their entire high school career. 


CONCEPTUAL FRAMEWORK 

This study draws conceptually on the work of Vygotsky (1978), most specifically on the notion 
that tools (in our instance computer software and hardware) impact on cognitive development. 

Mediation 

A fundamental premise of Vygotskian theory is that basic biological (or 'elementary') processes 
are transformed into higher cognitive functions through the use of culturally meaningful tools 
(such as language or, indeed a computer) during social interaction (Vygotsky 1978). That is, 
children are born with certain basic, biological processes, such as for example, perception and 
the potential for eidetic memory (Diaz, Neal and Amaya-Williams 1993). As the child develops 
within the social world, these elementary processes are transformed by the child's interaction with 
the social world. Higher cognitive functions develop first as interpsychological functions, with 
m/other initially guiding the child's activity, and later 'turn inward' becoming intrapsychological 
functions. Higher cognitive functions, then, have social origins. This conceptualisation of 
development famously overcomes the prior dualist 7 stance to development by positing that mind 
is social and is captured in Vygotsky's general genetic law: 

Every function in the child's cultural development appears twice: first, on the social level, 
and later on the individual level; first, between people (interpsychological), and then 
inside the child (intrapsychological). This applies equally to voluntary attention, to logical 
memory, and to the formulation of concepts. All the higher functions originate as actual 
relations between human individuals. Vygotsky 1978, p.57. 

What emerges from this law is a clear understanding that the nature and the quality of mediation 
are crucial in the development of higher cognitive functioning and, relatedly, self-regulation. 
Computer software, then, can be seen to mediate a student's access to mathematics, which is 
the hypothesis informing this research. 


7 That is, that mind is either naturally ‘given’ or socially derived. 
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RESEARCH DESIGN AND METHODOLOGY 
The Research Questions 

The following questions were investigated in this study: 

i. Are Matric Mathematics results in EMDC East high schools that have Khanya computers 
better than those at EMDC East high schools without Khanya computers? 

ii. Have Matric Mathematics results in EMDC East high schools improved since the 
beginning of the Khanya intervention? 

iii. Did the Khanya intervention result in a higher pass rate in Mathematics in EMDC East 
high schools? 

iv. Did the Khanya intervention result in a higher percentage enrolment in Higher Grade 
mathematics in EMDC East high schools? 

In order to establish the impact of technology on mathematical performance in sampled schools, 
the study undertook a quantitative analysis by comparing the Matric Mathematics results and 
enrolment of various schools in the EMDC East region; the latter chosen randomly from the four 
education districts in the greater Cape Town area. It is likely that the results of schools in this 
region are indicative and typical of all urban schools in the greater Cape Town area. 

The data used were secondary in nature; data already collected and categorised and accessed 
via the WCED database and Khanya. The WCED data provided the Matric Mathematics results 
(by symbol) for each school in the EMDC East, divided into Higher Grade and Standard Grade 
results. The Khanya data included a list of all the EMDC East schools, the Khanya wave to which 
each belonged, and the installation dates of the Khanya computer laboratories and software 

Sample 

31 high schools in the EMDC East region of Cape Town formed the sample. The experimental 
group consisted of all 14 EMDC East high schools who had had computers since at least 2001. 
That is, this represents the entire population of schools with the Khanya intervention. The control 
group was all 17 EMDC East high schools in the Khanya project who had access to computer 
from at least 2006.AII schools were chosen based on the assumption that computers are used for 
no less than 1 hour per week by the students for mathematics instruction and that the teachers 
have had at least on hour’s training in the requisite programmes (van Wyk, 2002). However, the 
authors understand that access to computers does not necessarily ensure use and we read our 
findings with this in mind. 

Data Analysis and Interpretation 

It is important to note that the logic behind the Khanya intervention, as stated by Louw et al. 
(2008), is this: “the principal cause of the low achievement levels in Mathematics was assumed to 
be the low capacity of teachers, and the ICTs would compensate for low-capacity teachers” (p. 
43). Put another way, “the Khanya computers and software were expected to provide the 
coverage of the curriculum that poorly trained teachers were not able to provide” (Louw et al., 
2008 p. 43). The Khanya project was, therefore, implemented in order to impact positively on 
students' mathematics results through the provision of computers and mathematics software. 

Louw et al's (2008) study, based on results in the 2003 Grade 12 final examination, provide a 
qualified ‘yes' as an answer to the question as to whether the Khanya intervention has actually 
succeeded in improving mathematics marks. The statistical analysis reported here, based on 
more recent data - the 2007 Matric results - represents an attempt to re-answer this question, 
and look at various new questions not answered by Louw et al. (2008), such as whether the 
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intervention resulted in an increase in Higher Grade (as opposed to Standard Grade) 
Mathematics enrolment. 


THE TESTS 

The Mean Student Score 

Matric results are given as symbols, is converted to numerical data for statistical analysis. The 
points allocation used by the University of Cape Town (UCT) for Matrics who wrote before 2008 
was selected as a suitable conversion table (see Table 1 below): 

Table 1: UCT’s admission rating system for the South African School Leaving 
Qualification 


ACADEMIC 

LEVEL 

ADMISSION POINTS BY SYMBOL 


A 

B 

C 

D 

E 

F 

Higher Grade 

8 

7 

6 

5 

4 

3 

Standard Grade 

6 

5 

4 

3 

2 

1 


In the Matric HG results received from Khanya and used for analysis no ‘F grade' totals are 
indicated. Instead, all F grades are grouped under ‘fail’ (as an F is indeed a higher grade fail). 
However, this did not provide a problem with this analysis as this (minor) absence is consistent 
across all the data. 

After using the above table to convert grades to points, an average score for each pupil was 
generated. The method used to generate this is quite simple: for each year and each wave we 
multiplied the number of A grades, B grades et cetera obtained by the pupils in each of the 
groups by the UCT points allocation. These were then summed and the total divided by the total 
number of students who wrote Matric Mathematics. We have termed this final statistic the ‘mean 
student score’ (MSS). 


Test 1: Comparing the 2007 Matric Mathematics Results of an Experimental and Control 
Group 

The first test involved comparing the 2007 Matric Mathematics results between two groups: an 
experimental group and a control group. The unit of analysis was individual schools, and the size 
of the study sample was 31 high schools. The experimental group consisted of all 14 EMDC East 
high schools in the Khanya Pilot, second and third waves 8 (and thus which received their Khanya 
labs from 2001 to 2003). Pupils at these schools would thus have had at least four years of 
access to the computer facilities, assuming they were used. The control group was all 17 EMDC 
East high schools in the Khanya sixth and seventh waves, which received their computer labs in 
the period between late 2005 and 2007. Pupils at these schools would thus have had little 
opportunity (on average around one year) to use the computer facilities. 


8 The different ‘waves’ mentioned refer to the different phases of implementation of the Khanya intervention, 
with the ‘pilot wave’ being the first group of schools that received the Khanya computers. 
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A t-test for independent samples was not able to be carried out as the Kolmogorov-Smirnov test 
for normality indicated that the data is not close to being normally distributed. 


Table 2: Mean Student Score Ranks (Test 1) 


TEST RESULTS 


Group 

N 

Mean Rank 

Sum of Ranks 

Mean Student Score 

2 control 

17 

14.35 

244.00 

3 experimental 

14 

18.00 

252.00 

Total 

31 




A non-parametric Mann-Whitney U test, which does not require the assumption of normality, was 
thus performed on the data instead. The test revealed no significant difference in the mean rank 
of the experimental group (mean rank = 18; n = 14) and the control group (mean rank = 14.35; n 
= 17), U = 91; z = -1.11; p = 0.266 (see Tables 2 above and 3 below). 


Table 3: Mann-Whitney U test results: mean student score (Test 1) 


TEST RESULTS 

Mann-Whitney U 

91.000 

Wilcoxon W 

244.000 

Z 

-1.111 

Asymp. Sig. (2-tailed) 

.266 

Exact Sig. [2*(1-tailed 

Sig.)] 

.279 a 

a. Not corrected for ties. 

b. Grouping Variable: Group 


The Mann-Whitney U test reported in table 3 revealed no significant difference in the mean rank 
of the experimental and the control group. The effect size statistic was also calculated as 0.2, 
which indicates a small to medium effect size using Cohen's (1988) criteria. In other words, a 
small to medium amount of the variance between the control and experimental groups' 
Mathematics results is explained by whether or not the students had access to computers. 

On re-analysis of the schools within each of the groups, it became clear that the MSS of four 
schools were acting as outliers, with MSS vastly superior to the other schools in the sample. The 
two groups were thus re-defined by excluding these schools; one because it was a selective 
intake school and three others because they are ex-Model C schools 9 with far superior facilities, 
better qualified teachers and a wealthier pupil body. The exclusion of these four schools ensured 
a more effective comparative dimension between the experimental and control schools. 


9 Model C schools were, during the Apartheid years, schools for White pupils only and were thus better 
resourced than township schools. 
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With these re-defined groups, the Kolmogorov-Smirnov test for normality indicated that the data 
were normally distributed. Levene's Test for the Equality of Variances gave a significance value 
of 0.4129, and thus we could assume equal variances for the two groups. 


Table 4: Group Statistics (Test 1 - redefined groups) 


TEST RESULTS 


N 

Mean 

Std. Deviation 

Std. Error Mean 

Mean Student Score 

e 

13 

.9286 

.48358 

.13412 

c 

14 

.6106 

.36505 

.09756 


A t-test for independent samples was performed with the re-defined groups to compare the mean 
student scores of the redefined control and experimental groups. This revealed that there is no 
significant difference between the results for the control schools (mean = 0.6106, std deviation = 
0.365) and the experimental schools (mean = 0.9286; std deviation = 0.484); t(25) = 1.938, p = 
0.064 (2-tailed) (see Tables 4 above and 5 below). 

The effect size statistic eta squared was calculated as 0.131, which indicates a small effect size 
using Cohen's (1988) criteria. In other words, only a small amount of the variance between the re¬ 
defined control and experimental groups' Mathematics results is explained by whether or not the 
students had access to computers. This is an incredibly low mean as some learners wrote 
standard grade mathematics and obtained a failing grade, which scored 0 points. 


Table 5: Independent Samples t-test results (Test 1 - redefined groups) 


TEST RESULTS 


Levene's Test 
for Equality 
of Variances 

t-test for Equality of Means 




95% Confidence 
Interval of the 
Difference 



Sig. 

t 

df 

Sig. 

(2-tailed) 

Mean 

Difference 

Std. Error 
Difference 

Lower 

Upper 

Mean 

Student 

Score 

Equal 

variances 

assumed 


.419 

1.938 

25 


.31806 



.65605 

Equal 

variances 

not 

assumed 



1.918 

22.297 

.068 

.31806 



.66175 











































Performance of school leavers in the Western Cape Province of South Africa 31 


Test 2: Comparing Matric Mathematics Results Before and After the Khanya Intervention 

The second test involved comparing the 2003 and 2007 Matric Mathematics results for Khanya 
schools in the fourth and fifth waves (a sample of 11 different schools). These 11 schools 
represent the entire population of schools which had long term exposure to computers. Schools in 
these two waves received their Khanya labs and software in the period from 2004 to mid 2005. 
Essentially this test enabled a comparison of results before and after the Khanya intervention, 
since in 2003 none of these schools had the Khanya facilities, and by 2007 they would have all 
had them for at least 2!4 years. 


Table 6: Paired Sample Statistics (Test 2) 


TEST RESULTS 


Mean 

N 

Std. Deviation 

Std. Error Mean 

Mean Student Score 
(before Khanya) 

.9549 

11 

.38195 

.11516 

Mean Student Score 
(after Khanya) 

.8267 

11 

.48608 

.14656 


The Kolmogorov-Smirnov test for normality indicated that the data were normally distributed, thus 
a paired samples t-test was conducted to evaluate the impact of the Khanya intervention on the 
MSS. There is no statistically significant change in the MSS from before Khanya (mean = 0.955; 
std deviation = 0.382) to after Khanya (mean = 0.827; std deviation = 0.486), t(10) = 0.958, p = 
0.361) (see Tables 6 above and 7 below). 


Table 7: Paired Samples t-test results (Test 2) 


TEST RESULTS 


Paired Differences 

t 

df 

Sig. (2- 
tailed) 

Mean 

Std. 

Deviation 

Std. Error 
Mean 

95% Confidence 

Interval of the 

Difference 

Lower 

Upper 

Mean Student 
Score (before 
Khanya)- 
Mean Student 
Score (after 
Khanya) 

.12818 

.44366 

.13377 

-.16987 

.42624 

.958 

10 

.361 


The effect size statistic eta squared was calculated as 0.084, which indicates a very small effect 
size using Cohen's (1988) criteria. Thus, only a very small amount of the variance between the 
Mathematics results pre- and post- Khanya intervention is explained by whether or not the 
students had access to computers. 
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Further Tests: Impact on the Matric Mathematics Pass Rate and Higher Grade Enrolment 

The above tests have indicated that the Khanya intervention has not brought about a significant 
improvement in the overall Matric Mathematics results. There are other questions that could then 
be asked, however, such as: 

• Did the Khanya intervention at least ensure a greater pass rate at Matric Mathematics? If 
this were true due to the intervention, it would be an important finding, and would indicate 
that the computers have been a success at improving the grades of the lowest achievers. 

• Did the Khanya intervention bring about a greater (percentage) enrolment in Higher 
Grade rather than Standard Grade Mathematics? Again, if this were true it would be most 
encouraging as it is indeed the stated desire of education authorities to have more pupils 
sit the exams at the former rather than the latter level, as Higher Grade Mathematics is 
one of the key requirements for entrance to critical university courses like engineering. 

Both these questions were answered by using quantitative measures. 

With regard to the first question above, for each school in the same sample group as for Test 2, 
the total number of passes at both Higher Grade and Standard Grade level, and the total number 
of Matric Mathematics candidates, was determined for both the years of 2003 and 2007. These 
figures were then used to calculate a percentage of pass for each school. This was the raw data 
on which a paired samples t-test was carried out to evaluate the impact of the Khanya 
intervention on the percentage of pupils passing Mathematics at Matric level (the Kolmogorov- 
Smirnoff test for normality showed that the data was indeed normally distributed). 


Table 8: Paired Samples Statistics (Test 3) 


TEST RESULTS 


Mean 

N 

Std. Deviation 

Std. Error Mean 

% passing (before 
Khanya) 

40.91 

11 

16.32 

4.92 

% passing (after 

Khanya) 

35.80 

11 

17.40 

5.25 


The results of this test showed that there was no statistically significant change in the pass 
percentage from before Khanya (mean = 40.9; std deviation = 16.3) to after Khanya (mean = 
35.8; std deviation = 17.4), t(10) = 0.878, p = 0.401 (see Tables 8 and 9). 
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Table 9: Paired Samples t-test results (Test 3) 


TEST RESULTS 


Paired Differences 





Mean 

Std. 

Deviation 

Std. Error 
Mean 

95% Confidence Interval 
of the Difference 

t 

df 

Sig. 

(2-tailed) 



Lower 

Upper 




% passing 
(before 
Khanya)- 
% passing 
(after 
Khanya) 

5.115 

19.324 

.5.826 

-7.867 

18.096 

.878 

10 

.401 


Table 10: Wilcoxon Signed Rank Test - Ranks (Test 4) 


TEST RESULTS 



N 

Mean Rank 

Sum of Ranks 

% on HG (after Khanya) - 
% on HG (before 
Khanya) 

Negative Ranks 

2 a 

2.50 

5.00 

Positive Ranks 

4 b 

4.00 

16.00 

Ties 

5 C 



Total 

11 



a. % on HG (after Khanya) < % on HG (before Khanya) 

b. % on HG (after Khanya) > % on HG (before Khanya) 


Table 10 responds to the question of whether the Khanya intervention caused an increase in the 
number of Matric Higher Grade Mathematics candidates; for each school in the same sample 
group as for Test 2, the number of higher grade candidates and total candidates for both 2003 
and 2007 was determined. This was then converted to a percentage; used as the raw data for the 
tests. 


Table 11: Wilcoxon Signed Rank Test Results (Test 4) 


TEST RESULTS 


% on HG (after Khanya) - % on HG (before 
Khanya) 

Z 

-1.153 a 

Asymp. Sig. (2-tailed) 

.249 

a. Based on negative ranks. 


The Kolmogorov-Smirnoff test for normality showed clearly that the data was not normally 
distributed, thus the non-parametric alternative to the paired sample t-test, the Wilcoxon Signed 
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Rank Test, was carried out. This test revealed no significant difference in the percentage of pupils 
enrolled in Higher Grade Mathematics after the Khanya intervention compared with before, with Z 
= -1.153 and p = 0.249 (see table 11). The median score of percentage HG enrolment did not 
change from pre-intervention to post-intervention, staying at 0% (see Table 12). 


Table12: WUcoxon Signed Rank Test - Descriptive Statistics (Test 4) 


TEST RESULTS 


N 

Percentiles 

25th 

50th (Median) 

75th 

% on HG (before 

Khanya) 

11 

.0000 

.0000 

2.2500 

% on HG (after Khanya) 

11 

.0000 

.0000 

8.9300 


The effect size statistic was also calculated as 0.25, which indicates a small to medium effect size 
using Cohen's (1988) criteria. In other words, a small to medium amount of the variance between 
the percentage of pupils enrolled in Higher Grade Mathematics is explained by whether or not the 
students had access to computers 


Interpretation of the Test Results 

Results from these tests appear to indicate that the Khanya intervention has not brought about a 
significant improvement in the sample schools’ Matric Mathematics results. In addition, 
calculations of effect score statistics showed that any variances in mean student scores were 
typically influenced in only a small way by the Khanya intervention. 

In fact, if one looks at the mean student score of the schools in the sample used for the second 
test (pre- and post-intervention), one can see that after the Khanya intervention the MSS has 
actually decreased (from 0.955 to 0.827, a decrease of 13.4% - see Table 6. Similarly, the 
percentage of pupils passing Mathematics at Matric level has also decreased after the Khanya 
intervention, from 40.9% to 35.8%. 

One needs to interpret these observations carefully, however, as it is not totally right to infer from 
this that the Khanya intervention has not had a cognitive impact. This is because, firstly, the 
statistical analyses showed that there was no statistically significant change in the pre- and post¬ 
intervention results in either direction, and secondly, the Khanya intervention, by which access to 
computers is enabled, is only one of the many factors that influence Matric Mathematics results, 
as is shown in the next section. 


Other factors influencing Mathematics results 

Taylor et al. (2003) have summarised the many factors which will influence pupil performance in 
South Africa. These include a number of factors - such as race, gender, socio-economic status 
and teacher-pupil ratios - which, whilst extremely significant in influencing Matric Mathematics 
results, would probably not be relevant at this instance as there would almost certainly be only a 
minimal change in these over the 4 year period (2003 to 2007) between the pre-Khanya results 
and post-Khanya results used in the above analysis. 
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However, the following influential factors might well have changed in the sample over the 4 year 
period, mainly due to the inevitable turnover of staff at schools: 

• Teacher qualifications 

• The teaching method utilised by the teachers 

• Availability and variety of learning materials 

• School ethos - particularly the presence of a joint vision between staff and pupils 
regarding the future of the school and the importance of a strong work ethic 

• The level of effectiveness of the school management 

• The level of discipline of pupils and teachers 

• Community relations - whether or not the students, staff and parents are working 
together to facilitate good education 

It might be that in a number of the schools in the EMDC East that were tested pre- and post- 
Khanya intervention there has been a decline in the quality of some or all of these measures 
listed above. If that were the case, it would certainly explain why the MSS has declined. Many of 
these variables are very difficult to measure (especially retrospectively) and, where they are 
quantifiable, access to such data will be extremely difficult due to their sensitive nature. This 
makes controlling (statistically) for these factors very difficult. In addition, the samples used in 
these tests are simply not large enough to attempt such highly complex models. 

The consequence of this is that we have not attempted to work any of these factors into our 
current analyses. However, their influence is large and could provide excellent research 
opportunities for those interested in pursuing this line of investigation. 

One other factor which was not mentioned by Taylor et al. (2003) but is obviously significant in 
the context of determining whether or not the use of computers has made a difference to 
Mathematics results, is the frequency of use of the mathematics software. Louw et al. (2008) 
performed correlational analyses on the relationship between improvement in Mathematics 
performance and the amount of time spent on the MasterMaths system, and found it to be 
positive, statistically significant and moderate in strength (r = 0.37; n = 125; p < 0.001). . 

Unfortunately, as has been shown by the same Louw et al. (2008) study into the use of 
MasterMaths in Khanya schools, pupils spent very little time using the software provided. In three 
of the experimental schools used in the study for which log files of MasterMaths usage were 
available, over a six month period Matric pupils logged onto MasterMaths an average of only 
seven times, for an average total of little over 214 hours (158 minutes). This raises the critical 
question as to how effective an intervention like Khanya can be if the advanced technology it 
provides is used so seldom. As Louw et al. (2008) state: “the statistics reported...are so low as to 
raise serious concerns about the implementation of the intervention” (p. 45). 


DISCUSSION AND CONCLUSION 

Our findings regarding the impact of computers on academic performance in Mathematics are 
uniform. Whether one is looking at a before- and after-scenario regarding the availability of 
computers, or a comparison between schools with and others without computers; in neither case 
do our findings shown a significant change in Matric Mathematics results. Similarly, no significant 
changes were shown in the percentage of passes, nor in the percentage of Higher Grade 
candidates, before and after the Khanya intervention. We began this paper posing the following 
questions: 

i. Are Matric Mathematics results in EMDC East high schools that have Khanya computers 
better than those at EMDC East high schools without Khanya computers? 
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ii. Have Matric Mathematics results in EMDC East high schools improved since the 
beginning of the Khanya intervention? 

iii. Did the Khanya intervention result in a higher pass rate in Mathematics in EMDC East 
high schools? 

iv. Did the Khanya intervention result in a higher percentage enrolment in Higher Grade 
mathematics in EMDC East high schools? 

Our findings reported above point to a negative response to all questions posed. 

These findings contrast with the majority of previous studies which found a positive, beneficial 
relationship between the use of computers and Mathematics results - for example, the studies of 
Christmann et al. (1997), Waxman et al. (2002), Banerjee et al. (2005) and Harrison et al. (2004). 
Our findings were more in line with the minority group that did not find a positive impact on the 
same, such as Angrist and Lavy (2002), and Wong and Evans (2007). Some major meta¬ 
analyses of computers’ impacts, such as those of Higgins (2001) and Tienken and Wilson (2007), 
agreed that whilst some studies have shown positive impacts other have shown otherwise. 

It needs to be re-iterated, however, that this study does not attempt to isolate the numerous 
factors that impact on Mathematics attainment in Khanya schools; instead focussing on only one 
(the provision of computers and Mathematical software). The fact that providing the facilities has 
not, in this instance, brought about an improvement in Matric Mathematics results could merely 
indicate that this intervention is insufficient to make a difference in isolation. Put another way, the 
other factors that are impeding the improvement in Mathematics results may be too strong to be 
overcome by this initiative alone. If and when these other impediments are overcome sufficiently 
then it is possible that the Khanya computers may prove their ability to impact Mathematics 
results positively. 

Further to this, this research did not include a quantitative analysis of the number of hours that 
the computers were utilised by each of the schools included in the various samples used in the 
tests. There were limited evidence that is available to answer this question - based on phone calls 
to around a dozen or so disadvantaged schools in the Cape Town area; the data on computer 
usage from a Khayelitsha case study school; and the work of Louw et al. (2008) in the same city - 
would seem to indicate that various factors have restricted the use of the Khanya computers in 
the mathematics class to, in most cases, seldom or never. Obviously, the mere presence of 
computers in these schools is not going to be enough to bring about an improvement in 
Mathematics grades; they need to be utilised for the purposes envisaged by the Khanya Project 
to ensure a positive impact. 

As Banerjee et al’s (2005) very encouraging study of a computer-assisted learning intervention in 
Vadodara, India, showed, the key is making use of the computers that are already in the schools 
but are not being used. “The programme found a way to make these computers pedagogically 
useful in the treatment schools, without placing additional demands on teachers’ time. It is the 
utilisation in this specific way and not the possession of the computers that had an impact” 
(Banerjee et al., 2005 p. 6). 

In conclusion, this research points to the fact that computers, alone, cannot impact on 
mathematical attainment; how they are used is critical to understand how they can potentially 
impact attainment. That is, further, qualitative work needs to be done to ascertain how or indeed 
whether, computers in South African schools are used as cognitive tools. 
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